[sonic-utilities] Update submodule; Build and install as a Python 3 wheel #5926

jleveque · 2020-11-15T19:09:52Z

Submodule updates include the following commits:

src/sonic-utilities 9dc58ea...f9eb739 (18):

Remove unnecessary calls to str.encode() now that the package is Python 3; Fix deprecation warning ([Accton AS7816-64X] Add new platform and device for AS7816-64X. #1260)
[generate_dump] Ignoring file/directory not found Errors ([Makefile]: variable ENABLE_SYNCD_RPC is always empty string #1201)
Fixed porstat rate and util issues ([Arista7260cx3] Increase /var/log partition size to 4G #1140)
fix error: interface counters is mismatch after warm-reboot ([build_debian] pip install pexpect in the base image for ansible module #1099)
Remove unnecessary calls to str.decode() now that the package is Python 3 ([build] allow user to override the default number of build jobs #1255)
[acl-loader] Make list sorting compliant with Python 3 ([build] make second Accton Debian package extra package of the first one #1257)
Replace hard-coded fast-reboot with variable. And some typo corrections ([DHCP relay]: Add patch to always undef VLAN_TCI_PRESENT so as not to treat VLAN-tagged packets differently #1254)
[configlet][portconfig] Remove calls to dict.has_key() which is not available in Python 3 (Per Vlan DHCP relay is broken if DHCP server is connected via Vlan interface #1247)
Remove unnecessary conversions to list() and calls to dict.keys() ([broadcom]: update sai to 3.0.3.3-1 #1243)
Clean up LGTM alerts ([Dell S6100]: EEPROM plugin overrides serial_number_str() to return service tag instead of serial number #1239)
Add 'requests' as install dependency in setup.py (Add caclmgrd and related files to translate and install control plane ACL rules #1240)
Convert to Python 3 ([interfaces.j2] Get mtu value from config DB if provided #1128)
Fix mock SonicV2Connector in python3: use decode_responses mode so caller code will be the same as python2 ([TACACS+]: Add support to specify source address for TACACS+ #1238)
[tests] Do not trim from PATH if we did not append to it; Clean up/fix shebangs in scripts ([DHCP relay]: Fix bug which could cause incorrect interface name association #1233)
Updates to bgp config and show commands with BGP_INTERNAL_NEIGHBOR table ([DHCP relay]: Fix bug which could cause incorrect interface name association #1224)
[cli]: NAT show commands newline issue after migrated to Python3 ([LLDP] Send interface name instead of mac as PORT_ID for mgmt interface #1204)
[doc]: Update Command-Reference.md ([Dell]: Change Dell psuutil.py file to work with Python 3.* #1231)
Added 'import sys' in feature.py file ([snmpagent]: Update sonic-snmpagent submodule #1232)
src/sonic-py-swsssdk 9d9f0c6...1664be9 (2):

Fix: no need to decode() after redis client scan, so it will work for both python2 and python3 (Update mellanox FW version #96)
FieldValueMap contains(in) will also work when migrated to libswsscommon(C++ with SWIG wrapper) (test new pr build, do not merge #94)

Also fix Python 3-related issues:
- Use integer (floor) division in config_samples.py (sonic-config-engine)
- Replace print statement with print function in eeprom.py plugin for x86_64-kvm_x86_64-r0 platform
- Update all platform plugins to be compatible with both Python 2 and Python 3
- Remove shebangs from plugins files which are not intended to be executable
- Replace tabs with spaces in Python plugin files and fix alignment, because Python 3 is more strict
- Remove trailing whitespace from plugins files

jleveque · 2020-11-18T00:56:38Z

@tahmed-dev: Please review. A few fixes were needed in sonic-config-engine.

lguohan · 2020-11-18T09:39:46Z

looks like error for load_minigraph, can you check?

01:26:20 TASK [execute cli "config load_minigraph -y" to apply new minigraph] ***********
01:26:20 Wednesday 18 November 2020 09:26:20 +0000 (0:00:00.062) 0:00:23.834 ****
01:26:48 fatal: [vlab-01]: FAILED! => {"changed": true, "cmd": "config load_minigraph -y", "delta": "0:00:25.419355", "end": "2020-11-18 09:26:46.044793", "msg": "non-zero return code", "rc": 1, "start": "2020-11-18 09:26:20.625438", "stderr": "Warning: Stopping telemetry.service, but it can still be activated by:\n telemetry.timer\nTraceback (most recent call last):\n File "/usr/local/bin/sonic-cfggen", line 432, in \n main()\n File "/usr/local/bin/sonic-cfggen", line 361, in main\n deep_update(data, FormatConverter.db_to_output(configdb.get_config()))\n File "/usr/local/lib/python3.7/dist-packages/swsssdk/configdb.py", line 443, in get_config\n cur = self.__get_config(client, pipe, data, 0)\n File "/usr/local/lib/python3.7/dist-packages/swsssdk/configdb.py", line 416, in __get_config\n keys = [key.decode() for key in keys if key != self.INIT_INDICATOR]\n File "/usr/local/lib/python3.7/dist-packages/swsssdk/configdb.py", line 416, in \n keys = [key.decode() for key in keys if key != self.INIT_INDICATOR]\nAttributeError: 'str' object has no attribute 'decode'", "stderr_lines": ["Warning: Stopping telemetry.service, but it can still be activated by:", " telemetry.timer", "Traceback (most recent call last):", " File "/usr/local/bin/sonic-cfggen", line 432, in ", " main()", " File "/usr/local/bin/sonic-cfggen", line 361, in main", " deep_update(data, FormatConverter.db_to_output(configdb.get_config()))", " File "/usr/local/lib/python3.7/dist-packages/swsssdk/configdb.py", line 443, in get_config", " cur = self.__get_config(client, pipe, data, 0)", " File "/usr/local/lib/python3.7/dist-packages/swsssdk/configdb.py", line 416, in __get_config", " keys = [key.decode() for key in keys if key != self.INIT_INDICATOR]", " File "/usr/local/lib/python3.7/dist-packages/swsssdk/configdb.py", line 416, in ", " keys = [key.decode() for key in keys if key != self.INIT_INDICATOR]", "AttributeError: 'str' object has no attribute 'decode'"], "stdout": "Executing stop of service telemetry...\nExecuting stop of service swss...\nExecuting stop of service lldp...\nExecuting stop of service pmon...\nExecuting stop of service bgp...\nRunning command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json --write-to-db\nRunning command: pfcwd start_default\nRunning command: /usr/local/bin/sonic-cfggen -d -y /etc/sonic/sonic_version.yml -t /usr/share/sonic/templates/sonic-environment.j2,/etc/sonic/sonic-environment", "stdout_lines": ["Executing stop of service telemetry...", "Executing stop of service swss...", "Executing stop of service lldp...", "Executing stop of service pmon...", "Executing stop of service bgp...", "Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json --write-to-db", "Running command: pfcwd start_default", "Running command: /usr/local/bin/sonic-cfggen -d -y /etc/sonic/sonic_version.yml -t /usr/share/sonic/templates/sonic-environment.j2,/etc/sonic/sonic-environment"]}

jleveque · 2020-11-18T09:40:20Z

@qiluo-msft: vsimage build is failing with the following swsssdk error. Can you please take a look?

Traceback (most recent call last):
  File "/usr/local/bin/sonic-cfggen", line 432, in <module>
    main()
  File "/usr/local/bin/sonic-cfggen", line 361, in main
    deep_update(data, FormatConverter.db_to_output(configdb.get_config()))
  File "/usr/local/lib/python3.7/dist-packages/swsssdk/configdb.py", line 443, in get_config
    cur = self.__get_config(client, pipe, data, 0)
  File "/usr/local/lib/python3.7/dist-packages/swsssdk/configdb.py", line 416, in __get_config
    keys = [key.decode() for key in keys if key != self.INIT_INDICATOR]
  File "/usr/local/lib/python3.7/dist-packages/swsssdk/configdb.py", line 416, in <listcomp>
    keys = [key.decode() for key in keys if key != self.INIT_INDICATOR]
AttributeError: 'str' object has no attribute 'decode'

lguohan · 2020-11-18T09:41:41Z

must be some python3 issue, str does not have decode method

jleveque · 2020-11-18T09:42:45Z

must be some python3 issue, str does not have decode method

Yes. It appears to be a Python 3 swsssdk error.

src/sonic-config-engine/config_samples.py

qiluo-msft · 2020-11-19T19:44:23Z

AttributeError: 'str' object has no attribute 'decode'

It is fixed by sonic-net/sonic-py-swsssdk#96

In Python 3, sort() takes no positional arguments, only keyword arguments. Fix acl-loader to comply. Fixes vsimage build failure in sonic-net/sonic-buildimage#5926

This reverts commit 1a3233c106c4028f2c23bdbe145b02d196906ab2.

…m.py

* src/sonic-utilities f46c27e...9d55082 (3): > [configlet][portconfig] Remove calls to dict.has_key() which is not available in Python 3 (#1247) > Remove unnecessary conversions to list() and calls to dict.keys() (#1243) > Clean up LGTM alerts (#1239)

…nd VS container

…de-syseeprom

lgtm-com · 2020-11-25T05:59:59Z

This pull request fixes 69 alerts when merging 3fab9f4 into dc15fbc - view on LGTM.com

fixed alerts:

63 for Unused import
4 for Result of integer division may be truncated
2 for Unused local variable

…heel (sonic-net#5926) Submodule updates include the following commits: * src/sonic-utilities 9dc58ea...f9eb739 (18): > Remove unnecessary calls to str.encode() now that the package is Python 3; Fix deprecation warning (sonic-net#1260) > [generate_dump] Ignoring file/directory not found Errors (sonic-net#1201) > Fixed porstat rate and util issues (sonic-net#1140) > fix error: interface counters is mismatch after warm-reboot (sonic-net#1099) > Remove unnecessary calls to str.decode() now that the package is Python 3 (sonic-net#1255) > [acl-loader] Make list sorting compliant with Python 3 (sonic-net#1257) > Replace hard-coded fast-reboot with variable. And some typo corrections (sonic-net#1254) > [configlet][portconfig] Remove calls to dict.has_key() which is not available in Python 3 (sonic-net#1247) > Remove unnecessary conversions to list() and calls to dict.keys() (sonic-net#1243) > Clean up LGTM alerts (sonic-net#1239) > Add 'requests' as install dependency in setup.py (sonic-net#1240) > Convert to Python 3 (sonic-net#1128) > Fix mock SonicV2Connector in python3: use decode_responses mode so caller code will be the same as python2 (sonic-net#1238) > [tests] Do not trim from PATH if we did not append to it; Clean up/fix shebangs in scripts (sonic-net#1233) > Updates to bgp config and show commands with BGP_INTERNAL_NEIGHBOR table (sonic-net#1224) > [cli]: NAT show commands newline issue after migrated to Python3 (sonic-net#1204) > [doc]: Update Command-Reference.md (sonic-net#1231) > Added 'import sys' in feature.py file (sonic-net#1232) * src/sonic-py-swsssdk 9d9f0c6...1664be9 (2): > Fix: no need to decode() after redis client scan, so it will work for both python2 and python3 (sonic-net#96) > FieldValueMap `contains`(`in`) will also work when migrated to libswsscommon(C++ with SWIG wrapper) (sonic-net#94) - Also fix Python 3-related issues: - Use integer (floor) division in config_samples.py (sonic-config-engine) - Replace print statement with print function in eeprom.py plugin for x86_64-kvm_x86_64-r0 platform - Update all platform plugins to be compatible with both Python 2 and Python 3 - Remove shebangs from plugins files which are not intended to be executable - Replace tabs with spaces in Python plugin files and fix alignment, because Python 3 is more strict - Remove trailing whitespace from plugins files

In Python 3, sort() takes no positional arguments, only keyword arguments. Fix acl-loader to comply. Fixes vsimage build failure in sonic-net/sonic-buildimage#5926

jleveque added Enhancement ➕ Submodule Update ⏫ labels Nov 15, 2020

jleveque self-assigned this Nov 15, 2020

jleveque changed the title ~~[sonic-utilities] Update submodule; Build and install sonic-utilities as a Python 3 wheel~~ [sonic-utilities] Update submodule; Build and install as a Python 3 wheel Nov 16, 2020

lguohan previously approved these changes Nov 16, 2020

View reviewed changes

jleveque dismissed lguohan’s stale review via f6707a7 November 16, 2020 17:42

jleveque mentioned this pull request Nov 17, 2020

Updating submodule pointers for sonic-utilities #5921

Merged

3 tasks

This comment has been minimized.

Sign in to view

jleveque requested a review from tahmed-dev November 18, 2020 00:56

jleveque mentioned this pull request Nov 18, 2020

Unable to install sonic_utilities-1.2-py3-none-any.whl with Build 492 #5942

Closed

lguohan previously approved these changes Nov 18, 2020

View reviewed changes

lguohan linked an issue Nov 18, 2020 that may be closed by this pull request

Unable to install sonic_utilities-1.2-py3-none-any.whl with Build 492 #5942

Closed

tahmed-dev previously approved these changes Nov 18, 2020

View reviewed changes

tahmed-dev reviewed Nov 18, 2020

View reviewed changes

src/sonic-config-engine/config_samples.py Show resolved Hide resolved

jleveque dismissed stale reviews from tahmed-dev and lguohan via fefcddf November 18, 2020 20:55

This comment has been minimized.

Sign in to view

jleveque added the Python 3 label Nov 19, 2020

jleveque mentioned this pull request Nov 20, 2020

[acl-loader] Make list sorting compliant with Python 3 sonic-net/sonic-utilities#1257

Merged

This comment has been minimized.

Sign in to view

jleveque mentioned this pull request Nov 21, 2020

Remove unnecessary calls to str.encode() now that the package is Python 3; Fix deprecation warning sonic-net/sonic-utilities#1260

Merged

jleveque added 22 commits November 25, 2020 05:45

Update submodule further

302255d

Add 'pyang' to install dependencies of sonic-yang-mgmt package

baf95b8

Install pyang in Buster slave container

170f683

Install Python 3 libyang package in VS container

37028c3

Install Python 3 sonic-config-engine package in host OS

40a115b

Revert "Add 'pyang' to install dependencies of sonic-yang-mgmt package"

093e1b9

This reverts commit 1a3233c106c4028f2c23bdbe145b02d196906ab2.

[sonic-confic-engine] Use integer division in config_samples.py

8f9ab66

[x86_64-kvm_x86_64-r0] Use Python 3-compliant print function in eepro…

25e5e55

…m.py

Update utilities further

53877c1

* src/sonic-utilities f46c27e...9d55082 (3): > [configlet][portconfig] Remove calls to dict.has_key() which is not available in Python 3 (#1247) > Remove unnecessary conversions to list() and calls to dict.keys() (#1243) > Clean up LGTM alerts (#1239)

Update swsssdk submodule

b757b0c

Explicitly install Python 3 pyangbind and uninstall enum 34 in host a…

e70daa2

…nd VS container

Update utils further

3c47a6a

Update utils further

576d9b8

Make all eeprom plugins Python 3-compliant to work with Python 3 deco…

c95f65d

…de-syseeprom

Update utils further

11887ec

Make all other platform plugins compliant with both Python 2 and 3

59ffeda

Update shebangs to Python 3, remove unnecessary ones

695a509

Replace tabs with spaces in Python plugins; fix alignment using autopep8

2e4475e

Format remaining Python files wil autopep8 for consistency

981c5af

Fix comment placement

9529029

Fix alignment

c313fa6

Pass 'universal_newlines=True' arg to all relevant subprocess functions

3fab9f4

jleveque dismissed tahmed-dev’s stale review via 3fab9f4 November 25, 2020 05:46

lguohan approved these changes Nov 25, 2020

View reviewed changes

jleveque merged commit 7f4ab8f into sonic-net:master Nov 25, 2020

jleveque deleted the utilities_py3 branch November 25, 2020 18:28

mudsut4ke mentioned this pull request Nov 27, 2020

[device/celestica]: Add xcvrd event support for Seastone-DX010 #5896

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[sonic-utilities] Update submodule; Build and install as a Python 3 wheel #5926

[sonic-utilities] Update submodule; Build and install as a Python 3 wheel #5926

jleveque commented Nov 15, 2020 •

edited

Loading

This comment has been minimized.

jleveque commented Nov 18, 2020

lguohan commented Nov 18, 2020

jleveque commented Nov 18, 2020

lguohan commented Nov 18, 2020

jleveque commented Nov 18, 2020

This comment has been minimized.

This comment has been minimized.

qiluo-msft commented Nov 19, 2020

This comment has been minimized.

This comment has been minimized.

lgtm-com bot commented Nov 25, 2020

[sonic-utilities] Update submodule; Build and install as a Python 3 wheel #5926

[sonic-utilities] Update submodule; Build and install as a Python 3 wheel #5926

Conversation

jleveque commented Nov 15, 2020 • edited Loading

This comment has been minimized.

jleveque commented Nov 18, 2020

lguohan commented Nov 18, 2020

jleveque commented Nov 18, 2020

lguohan commented Nov 18, 2020

jleveque commented Nov 18, 2020

This comment has been minimized.

This comment has been minimized.

qiluo-msft commented Nov 19, 2020

This comment has been minimized.

This comment has been minimized.

lgtm-com bot commented Nov 25, 2020

jleveque commented Nov 15, 2020 •

edited

Loading